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DETECTION AND REMOVAL OF BLEMISHES IN DIGITAL IMAGES UTILIZING 
ORIGINAL IMAGES OF DEFOCUSED SCENES 



CROSS-REFERENCE TO RELATED APPLICATIONS 

5 This application claims priority to United States provisional patent application 

60/773,714, filed February 14, 2006, and United States patent application no. 
1 1/673,577, filed February 10, 2007, both of which are incorporated by reference, 
and this application is a CIP of United States patent application serial no. 
10/676,823, filed September 30, 2003, published as 2005/0068446 on March 31, 
10 2005, which is incorporated by reference, and is related to a family of patent 
applications filed on the same day including 2005/0068450, 2005/0068451 , 
2005/0068452, 2005/0078173, 2005/0068448, 2005/0068445, and 2005/0068447, 
which are also incorporated by reference, as is US patent 6,035,072. 



15 BACKGROUND 

1. Field of the Invention 

This invention related to digital photography, and in particular, automated 
means of removing blemish artifacts from de-focused images captured and digitized 
in a digital process. 

20 

2. Description of the Related Art 

Many problems are caused by dust in particular and blemishes in general on 
imaging devices in general and digital imaging devices in particular. In the past, two 
distinct forms of image processing included providing means to detect and locate 

25 dust, scratches or similar defects and providing means to remedy the distortions 
caused by the defects to an image. It is desired to have an advantageous system 
that combines these functions, and can automatically detect and correct for the 
effects of dust, scratches and other optical blemishes. 

Dust has been a problem in scanning devices for a long time. Various 

30 aspects of the scanning process, enhanced by image processing techniques where 
appropriate, have been employed to provide means for the detection of dust or 
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defects relating to document or image/film scanners. These devices form an image 
by moving a 1D sensor pixel array across a document platen, or in some cases, the 
document platen with document/image, is moved across the sensor array. The 
physics and implementation of these devices differ significantly from those of a field- 
5 based sensor or camera-style device. It is desired particularly to have dust and/or 
blemish detection and correction techniques for field-based or camera-style 
acquisition devices. 

Image correction has been studied in relation to display devices, output 
apparatuses such as printers, and digital sensors. Image correction of dust artifacts 

10 can be used to recreate missing data, also referred to as in-painting or restoration, or 
undoing degradation of data, which still remains in the image, also referred to as 
image enhancement. It is desired to have a system including a digital camera and 
an external device or apparatus that can facilitate a defect detection and/or 
correction process involving sophisticated and automated computerized 

15 programming techniques. 

SUMMARY OF THE INVENTION 

A digital image acquisition device is provided which includes an image 
acquisition sensor, a shutter, optical elements for focusing an image on the sensor, 

20 an aperture disposed between said optical elements and said shutter, a translucent 
lens cap which defocuses images when in place, and digitally-embedded 
programming. The device is arranged to automatically capture a calibration image of 
a de-focused scene when the shutter and the aperture are open. A map of defects is 
derived from the calibration image acquired by the sensor when illuminated by 

25 ambient light. 

The device may include a trigger for automatically acquiring a de-focused 
calibration image which is arranged to automatically detect whether the translucent 
lens cap is in place as a pre-requisite for triggering capture of the defocused 
calibration image. The trigger may be arranged to trigger image capture based at 

30 least in part on time interval, change of lens, or amount of images taken, or a 

combination thereof. The trigger may include programming for analyzing whether 
the image is sufficiently de-focused as a pre-requisite for capture of the de-focused 
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calibration image. The trigger may be arranged to initiate capture of a plurality of de- 
focused calibration images of the scene within a certain time frame. 

A further digital image acquisition device is provided which has no 
photographic film. The device includes an optical system, a processor and one or 
5 more storage devices for programming the processor to control the camera to 
perform a method of automatically correcting dust or other blemish artifact regions 
within acquired images. The method includes acquiring one or more original digital 
images through a translucent lens cap disposed along an optical path between an 
object and one or more components of the optical system including a digital sensor. 

10 Probabilities that certain pixels correspond to dust or other blemish artifact regions 
within the one or more original digital images are determined. The dust or other 
blemish artifact regions are associated with one or more extracted parameters 
relating to the optical system when the one or more original digital images were 
acquired. A statistical dust map is formed including mapped dust or other blemish 

15 regions based on the dust or other blemish artifact probability determining and 
associating. Pixels are corrected corresponding to dust artifact regions within the 
one or more original images based on the associated statistical dust map. 

The one or more original images may include original images of de-focused 
scenes. 

20 A plurality of individual probabilities may be statistically combined based on 

regions within two or more images. Probabilities may be determined that certain 
pixels correspond to regions free of dust or other blemishes within the images based 
at least in part on a comparison of suspected dust artifact regions within one or more 
of the images. Certain suspected dust or other blemish artifact regions may be 

25 eliminated as having a probability below a first threshold value. Certain probable 
dust or other blemish artifact regions may be judged as having a probability above a 
second threshold value such as to be subject to the correcting. 

The one or more extracted parameters may include aperture size or focal 
length or both. 

30 The method may further include digitally-acquiring additional images with the 

digital acquisition device, repeating the determining and associating, and updating 
the statistical dust map including updating the mapped dust or other blemish regions 
based on the additional dust or other blemish artifact determining and associating. 
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The determining may include dynamically updating the probabilities based on 
comparisons with suspected equivalent dust artifact regions within the additional 
digitally-acquired images. 

It may be validated whether additional digitally-acquired images have non- 
5 contradicting probability data that certain pixels correspond to dust or other blemish 
artifact regions within the additional digitally-acquired images prior to correcting 
pixels corresponding to correlated dust artifact regions within additional digitally- 
acquired images based on the associated statistical dust map. 

The dust or other blemish artifact determining may include: 
10 (I) loading the statistical dust map; 

(II) loading extracted parameter information of a present image; 

(III) performing calculations within the statistical dust map having extracted 
parameter variable-dependencies; and 

(IV) comparing dust artifact detection data with the extracted parameter 
15 dependent statistical dust map data. 

The dust or other blemish artifact determining may also include: 

(I) loading the statistical dust map; 

(II) loading extracted parameter information of a present image; 

(III) performing a calculation for relating the statistical dust map with the 
20 present image according to a selected value of an extracted parameter which is 

otherwise uncorrelated between the present image and the dust map; and 

(IV) comparing dust artifact detection data with the now correlated statistical 
dust map data. 

The method may further include sorting dust or other blemish artifact 
25 distribution data within the dust map according to meta-data. 

The digital acquisition device may capture the images from film images. 

A method is also provided for automatically correcting dust or other blemish 
artifacts within images acquired by a system including a digital acquisition device 
including a lens assembly is provided. The method includes acquiring multiple 
30 original digital images of a defocused scene with the digital acquisition device having 
a translucent lens cap in place. Probabilities that certain pixels correspond to dust or 
other blemish artifact regions within the images are determined based at least in part 
on a comparison of suspected blemish artifact regions within two or more of the 
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images. Probable blemish artifact regions are associated with one or more values of 
one or more extracted parameters relating to the lens assembly of the digital 
acquisition device when the images were acquired. A statistical dust or other 
blemish artifact map is formed including mapped blemish regions based on the dust 
5 or other blemish artifact determining and associating. Pixels corresponding to 

correlated dust artifact or other blemish regions are corrected within further digitally- 
acquired images based on the associated statistical dust map. 

The image correction method may be automatically performed within a digital 
camera that includes an optical system including one or more lenses, a shutter and a 

10 translucent lens cap, a sensor array, processing electronics, and a memory. The 
image correction method may also be performed at least in part within an external 
processing device that couples with a digital camera that includes the optical system 
and the sensor array to form a digital image acquisition and processing system that 
also includes the processing electronics and the memory. The programming 

15 instructions may be stored on a memory within a camera and/or the external device 
which performs the image correction method. The digital acquisition device may 
capture images from film images, and the digital acquisition device may include a 
digital camera. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a main workflow of a dust removal process in accordance 
with a preferred embodiment. 

Figure 2a illustrates the creation of a dust map. 

Figure 2b illustrates an alternative embodiment of the creation of a binary dust 

25 map. 

Figure 3 outlines a correlation of a dust map to image shooting parameters. 

Figure 4a illustrates a procedure for detecting and removing dust from images in 
accordance with a preferred embodiment. 

Figure 4b illustrates reconstruction in-painting of dust regions. 
30 Figure 4c illustrates the occurrence of dust in a picture with high edge 

information. 

Figure 4d illustrates the numerical gradient in an image. 



WO 2007/095483 



PCT/US2007/061964 



-6- 

Figures 4e-4f illustrate a spiral procedure of in-painting from the periphery 
inwards. 

Figure 5 illustrates a quality control method of checking the dust map. 

Figures 6a-6f represents an optical geometry of a lens that may be used in 
5 accordance with a preferred embodiment: 

Figures 6a, 6b and 6c represent three lenses with the same, fixed, focal 
distance, with the same focal number, but with different constructions. 

Figure 6d illustrates the concept of an exit pupil and the distance to it. 

Figure 6e illustrates intersections of the principal ray with windows and image 
10 plane, which is preferably approximately the sensor plane. 

Figure 6f illustrates the shift of one dust spec in comparison with a shift of 
another dust spec in an image. 

Figure 6g illustrates the capturing of images of de-focused scenes when a 
translucent lens cap is in place. 
15 Figures 7a-7f generally illustrate effects of dust on the creation of an image 

using a known optical system: 

Figure 7a illustrates an influence of a dust particle located on the input surface 
of the window on the beam converging towards the imaging point. 

Figure 7b illustrates a side view of the rays as obscured by dust as a function 
20 of the aperture. 

Figure 7c illustrates a frontal projection of the rays as obscured by a dust as a 
function of the aperture. 

Figure 7d and Figure 7e illustrate a power distribution map of the same dust 
spec as manifested in different f-stops. 
25 Figure 71 illustrates an effect of the change in focal length on the area 

obscured by the dust. 

Figure 8 illustrates an adjustment of the dust map based on aperture. 

Figure 9 illustrates an adjustment of a dust map based on focal length. 

Figure 10 illustrates a process of estimating based on empirical data the 
30 values of the parameters in the formulae that defines the change in dust as a 
function of change in focal length. 
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Figure 11 illustrates a process of estimating based on empirical data the 
values of the parameters in the formulae that defines the change in dust as a 
function of change in aperture. 

Figure 12 illustrates a further process in accordance with another 
5 embodiment. 



BRIEF DESCRIPTION OF TABLES 
Table 1 lists parameters in mathematical formulations of the optical system. 
Table 2 lists potential Extracted Lens Parameters. 

10 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



SOME DEFINITIONS 

Dust specs: The preferred embodiment takes advantage of the fact that many 
15 images may have repetitive manifestation of the same defects such as dust, dead- 
pixels, burnt pixels, scratches, or other blemishes. Collectively, for this specification 
all possible defects of this nature are referred to in this application as dust-specs or 
dust defects, even though such defects may be caused by other sources of 
blemishes. The effects of those dust specs on digital images are referred to herein 
20 as dust artifacts. 

Acquisition device: the acquisition device may be a multi-functional electronic 
appliance wherein one of the major functional capabilities of said appliance is that of 
a digital camera. Examples include digital camera, a hand held computer with an 
imaging sensor, a scanner, a hand-set phone, or another digital device with built in 
25 optics capable of acquiring images. Acquisition devices can also include film 
scanners with an area-capture CCD, as contrasted, e.g., with a line scan 
mechanism. 

D-SLR: Digital Single Lens Reflex Camera. A digital camera where the 
viewfinder is receiving the image from the same optical system as the sensor does. 
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Many D-SLR, as for SLR cameras have the capability to interchange its lenses, this 

exposing the inner regions of the camera to dust. 

A few parameters are defined as part of the process: 

N Number of images in a collection. 

Hidp Number of images for occurrence to be high dust probability. 

HSdp Number of recurring specs for label a region to be high dust 
probability. 

p(hsdp) Probability threshold for high confidence of a dust spec. 

N dp Number of images to determine that a region is not dust. 

p(ndp) Probability threshold to determine that a region is not a dust region. 



5 Most likely Hidp <= HSdp 

I - is a generic image. 
I(x,y) pixel in location x horizontal, y vertical of Image I. 

pM is a continuous tone, or a statistical representation of a dust map. 

dM is a binary dust map created form some thresholding of pM. 

MATHEMATICAL MODELING OF THE OPTICAL SYSTEM 
Prior to understanding the preferred embodiments described herein, it is 
helpful to understand the mathematical modeling of the camera optical system. With 
10 this modeling, the preferred embodiments may advantageously utilize a single dust 
map for dust detection and/or correction techniques rather than creating a dust map 
for each instance of dust artifact having its own shape and opacity depending on 
extracted parameters relating to the imaging acquisition process. With an ability to 
model the optical system and its variability, a single map may suffice for each lens or 
15 for multiple lenses, and for multiple focal lengths, multiple apertures, and/or other 
extracted parameters as described in more detail below. 

In order to study the shadow of an object on a plane, it is helpful to consider 
the following: 

- the illumination of the object (the spectral and coherent characteristics of 
20 the light) 

- the shape of the object (including its micro-geometry) 

- the reflection and transmission properties of the object 

- the relative position of the object and the plane 
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The case of interest for defining the model of dust as it appears on the sensor, 
is that of an object close to the image plane of a lens, which is a complex optical 
system, with natural light illumination. The object shape and its reflection and 
transmission properties are practical impossible to model because no any specific 
5 information on the dust particle is available. On the other side, because the dust 
particles are small, it is very probable to have the same reflection and transmission 
properties on their surfaces. The distance between the dust and the sensor, which is 
the image plane is small and is in the order of magnitude of fraction of a millimeter. 

Some Definitions as to notations are now provided: 



Pe- 


exit pupil position, the distance between the sensor, and the Exit 
Pupil 


tw - 


thickness of the window, distance of dust to the image plane 


f- 


focal length of the objective 


f/#- 


focal number 


(u,v) - 


denote the coordinate system of the input surface, having its origin 
in the intersection point with the lens optical axis. 


(x,y)- 


the dust position on the image plane in the coordinate system in the 
image plane with the origin in the intersection point with the lens 
optical axis. 


i(*,y)- 


be the value of the irradiance in the image point (x,y) 


D - 


is the exit pupil diameter 


ho 


is the distance of the objects on the dust plane to the center of the 
optical path 


h 


is the distance form the object to the center of the optical axis 


h k 


is the distance of dust spec k on the dust plane to the center of the 
optical path 



10 Table 1: Parameters in Mathematical Formulation of the Optical System 

Figure 6a, 6b and 6c represent three lenses with the same, fixed, focal 
distance, with the same focal number, but with different constructions. The first 
construction, in figure 6-a is the most common. The second in figure 6b is specific for 
15 the lenses used in metrology. This type of lens is called telecentric in the image 
space. The third in figure 6c construction is rarely found in optical systems, but not 
impossible. 

Figure 6d illustrates the concept of an exit pupil and the distance to it. The exit 
pupil distance, 644, is the parameter that defines this property, and represents the 
20 distance from the secondary principal plane, 646, of the lens and the intersection of 
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the axis of the light cone and optical axis of the lens. The focal distance 644 
represents the distance form the secondary principal plane to the object plane. 

In the case of the second type lens, as defined in Figure 6b, the exit pupil 
position is at infinity. In the first and third case, figures 6a and 6c respectively, the 
5 intersections of the light cone with the dust plane are ellipses. In the case of the 
second lens type, Figure 6b the intersection of the light cone with the dust plane is a 
circle. 

In the case of zoom lens, the exit pupil position can be constant (at infinity), or 
can vary significantly, depending on the construction of the lens. 
10 This information about the exit pupil is usually not public because, in general, 

it is not useful for the common user of photographic or TV lenses. However, this 
information can be readily available with some measurements on an optical bench. 
Alternatively, this information can be achieved based on analysis of dust in the 
image. 

15 Figure 6e illustrates the Intersections of the principal ray with windows and 

image plane, which is the sensor plane. This figure illustrates the various 
parameters as defined in Table 1 . Figure 6f illustrates the shift of one dust spec in 
comparison with a shift of another dust spec in an image. 

Figure 7a describes the influence of a dust particle (obscuration) located on 

20 the input surface of the window (obscuration plane), on the beam converging 
towards the imaging point. 

Figure 6g illustrates the capturing of images of de-focused scenes when a 
translucent lens cap is in place. With the lens cap removed, an object may be in 
focus at the image plane of a camera's detector D. With the translucent lens cap on, 

25 however, light rays traveling through the translucent lens cap are disrupted resulting 
in a de-focused image at the detector D. 

In the embodiment of Figure 6g, the camera used to capture the image is 
arranged to receive a translucent lens cap. The translucent lens cap serves to 
provide an ideal source of a defocused (smooth) image to the camera sensor D in 

30 order to enable the camera to capture an image of a defocused scene or object, 
thereby producing a simpler image or set of images to be referenced for dust map 
calculations. 
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In such an embodiment, the camera is arranged to automatically detect 
whether the lens cap is in place in order to trigger the capture of the defocused 
image. In such an embodiment, the dust map may be generated periodically and 
automatically, without the user having to perform any calibration action. The trigger 
5 to automatically acquire such image for the dust map can be time interval, change of 
lens, amount of images taken, replacement of lens cap, or combinations of these or 
other parameters. 

Alternatively, the camera may be arranged to detect if a lens cap is on prior to 
capturing an image and analyzing if the image captured is highly defocused. 

10 Alternatively, the camera is arranged to automatically acquire an image and 

determine if the image is of a generally de-focused scene. Alternatively, captured 
images may be examined to determine whether they are of a generally de-focused 
scene suitable for dust calibration. In any case, once the camera is imaging a 
defocused scene, the camera is arranged to capture a plurality of images of such a 

15 scene within a certain time frame. This embodiment may include a camera arranged 
to perform any combination of these de-focused image-capturing techniques. 

Figure 7b illustrates the side view of the rays as obscured by a dust as a 
function of the aperture. 

Figure 7c illustrates the frontal projection of Figure 7-b, of the rays as 

20 obscured by a dust as a function of the aperture. 

Figure 7d and Figure 7e illustrates a power distribution map of the same dust 
spec as manifested in different f-stops, namely a relative open aperture f-9 for Figure 
7d and a closed aperture, namely f-22 and higher, in Figure 7e. One can see that 
the dust spot as recorded with the closed aperture is much sharper and crisper. 

25 The dust particle is situated in the window plane, and totally absorbing or 

reflecting the portion of the beam (S2) that normally reaches the image point (P). 

Let l(x,y) be the value of the irradiance in the image point (x,y) in the absence 
of the dust particle. Assuming that the energy in the exit pupil of the optical system is 
uniformly distributed, then, in the presence of the dust particle, the l(x,y) will be 

30 reduced proportionally with the ratio between the area of the intersection of the light 
beam (S1 ) with the clean zone of the window, and the whole area of the same 
intersection (Sc). 
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Remember that Sc is a function of the lens f-number (f/#) and window 
thickness t w , so the value of the real irradiance in the presence of dust will be: 



(4) 



10 



or, by taking into account that S1 = Sc - S2 and S2 = area of intersection between 
the dust particle and the illuminated area: 



r(x,y)=l(x,yjl 



S2 

Sc 



i{xA\- 



Sc 



(4a) 



Because the dust particle is small, the next assumption can be made for all 
the image points affected by a specific dust particle, the area Sc remains constant. 
In the case of the telecentric illumination, this assumption holds true for all the 
15 image points. 

With the assumption above, we can correlate the dust geometry, position and 
shadow. 

First of all, we now study the distribution of the irradiance on the input plane of 
the window. The intersection of the conic beam with this plane is generally an ellipse. 
20 The major axis of this ellipse is along the intersection of the plane mentioned above 
and the plane determined by the image point and the optical axis of the lens. 

The minor axis is: 



25 whereas the major axis is: 



b = t 



D 



D 



a 



cos 



arctg | 



(5) 



e\J 



(6) 



30 where D is the exit pupil diameter, so: 



D = 



//# 



(7) 
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In order to reach a complete formula of the irradiance distribution we now 
stress the assumption that for all image points affected by a chosen dust particle, "a" 
varies insignificantly. Let (u 0 ,v 0 ) be the point of intersection of the principal ray with 
the plane mentioned above. 

So, the characteristic function of the illuminated zone will be: 



C = C(u 9 v 9 u Q9 v Q9 f\f/# 9 P e9 t w ) 



(8) 



Let D(u ,v) be the characteristic function of the dust Then, the center of the 
10 cone intersection with the input plane of the window will be given by (3). 
Thus, 



S ddc = \\ D(u 9 v)* C(u 9 v 9 u 09 v 09 f\ f /# 9 P e9 t w )dudv 



R xR 



(9) 



r(x,y)=l(x,y) 1- 



\\d(u, v)* C(u, v, u 0 ,v 0 ,f,f/#,P e ,t w )dudv 



RxR 



Sc 



(10) 



which eventually, using the relation (3), yields 

\\D{u,v)*C 



r(x,y)=l{x,y) 1- 



RxR 



f 


( t 1 




f 


o 






X, 


1 + 




V 


v p.) 






Pe) 



y,f\f/#,P e9 t M 



7.0 
dudv 



Sc 



(11) 



25 The terms (1+t w /P e )x and (1+t w /P e )y determine the "movement" of the dust shadow 
with the change of the focal distance and, implicitly, of the Pe, as explained later on 
in equation 13. 



In the case of the telecentric illumination: 
30 - Pe is infinite, 

- uO = x and v0=y. 

- the ellipse becomes a circle 
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Qualitatively, as described quantitatively in equation 1 1 , the fall off, or the 
diminished effect of the dust on an image varies as follow: The dust is becoming 
less significant as the aperture becomes larger, or the f-stop is smaller and the pixels 
are inversely affected based on the distance of the pixel to the periphery of the dust. 



based on the focal length of the lens. In this figure one can qualitatively see that the 
area covered by the dust spec will be shifted as a function of the focal length. 

Also, information about the spatial distribution and/or orientation of dust 
particles within the dust map may be used to assist with determining probabilities 

10 that regions are either dust or non-dust. As an example, one could expect that a 
global average of the orientation of elongated dust particles would average to zero. 
However, at a more localized level, charged and/or magnetized dust particles can 
tend to align with similar orientations due to local dielectric forces. This effect would 
be partly dependent on the size, overall shape distribution (round or more elongated) 

15 and the overall distribution (uniform or clumped) of the dust regions in the dust map. 
This can tend to be particularly helpful when a decision is to be made as to whether 
to eliminate certain regions, e.g., most of the dust may be aligned in a radial direction 
relative to the center of the sensor, and then particles in a concentric alignment are 
less likely to be dust. 

20 Given the definitions above, a more quantitative model can be deducted: 



25 because the ratio t w /P e is small. Note that Pe has in common cases a negative 
value. 

Let (u,v) denote the coordinate system of the input surface, having its origin in 
the intersection point with the lens optical axis. Similarly, let (x,y) be the coordinate 
system in the image plane with the origin in the intersection point with the lens 
30 optical axis. 

Thus, if the principal ray of the beam intersects the input window plane in the 
point {u,v), the point in the image plane on the same principal ray will be The inverse 
transformation of (2) 



5 



Figure 7f illustrates the adaptation between the dust map and the image 




(12) 



h = h Q 
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- 15 



0 



0 1 — 



(2) 



5 with the same assumption of a small tw/Pe ratio will be 



1 + ^- 0 

P e 

0 1 + ^ 



10 



(13) 



Figure 7j depicts how points will shift as a function of the proximity to the optical 
center. Basically the farther the dust is form the center, the larger the displacement 
is. However, the direction and size of the displacements can be estimated. At the 
15 same time, the closer the dust spot is to the center, the more its shape will change. 

There are a few parameters that are not published. The distance, Tw, of the 
dust to the image plane is typically fixed for the camera and a single measurement 
can achieve that. 

The exit pupil distance may vary depending on the focal length of the lens, in 
20 particular for a zoom lens, the Exit pupil distance may not be most likely fixed, and 
can vary up to a telecentric mode where the distance is infinity. 

However, such information can be empirically extracted by analysis of images 
taken by a known camera and lens, as illustrated below and in Figure 7e. In this 
case, knowledge of the effect of one dust spec can be used to calculate the shift for 
25 any other dust spec in the image. 

In this figure a hypothetic zoom lens of 70mm-210mm is illustrated: 

- Pe-70 is the exit pupil position for the lens at 70mm, 

Pe-210 is the exit pupil location when the lens is at its highest enlargement 
210mm. 

30 - k, is known dust 

- m is hypothetical dust 

- h k is h for a specific dust particle k. 
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Knowledge of the shift of dust image (preferable near to one of the corners of 
the image). 



The equation (12) 



h = h, 



f t ^ 

1 2. 



(12) 



can be written for a specific dust particle (k)and underline the dependence 
with the focal distance, 



10 



15 



hif) = Kk 



P e (f\ 



(12-a) 



or 



Kif) 



l 0k 



1- 



Peifl 



independent from k (1 2-b) 



So , if we know the evolution of the image of the "k" dust particle we can find the 
evolution of the "m" dust image 



20 



25 



h m {f) = ^h k (f) (12 - c) 

The problem with this formula is again we don't know the positions of the dust 
particles 

Writing the upper formula for a given focal distance, say f 0 ,we have 



'0m 



K(fo) 



So, finally 



K k = K{fo) (12 " d) 



30 This relation does not need the exit pupil dependence, the thickness of the window 
and the dust position. 
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Alternatively, given formula (13) and given an image where the dust can be detected, 
one can calculate Pe as follows: 

u v P e 

(13-a) 

5 or 

^=(i + 0*- = (i + 0*- 

x y 

(13-b) 



In summary of the mathematical model of the optical system for dust, in the general 
10 case, the way the image is affected by the dust depends on: 

- exit pupil position Pe 

- thickness of the window tw 

- focal length of the objective f 

- focal number f/# 
15 - the dust position on the image plane (x,y) 

This can be calculated by knowing the optical system's components: 

a) the window thickness (tw), 

b) the function Pe(f), and 

c) the coordinates (u,v) of the dust if we want to determine the dust 
20 image position on the image plane. 



The main workflow of detecting and removing the dust from an image is 
illustrated in Figure 1 . The preferred embodiment is activated in four different cases 
In general, this preferred embodiment works for removing dust form a collection of 
25 images having the same acquisition device. Specifically, a user may acquire a 

picture on her digital camera (as illustrated in Block 101). Alternatively (102), a user 
may open a single image on a external device such as a personal computer, open 
(103) a folder of images on an external device or open a collection of images on a 
digital printing device (104). 
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The preferred embodiment then extracts the shooting parameters (120). Such 
parameters include, but not limited to data about: Camera name, Lens brand, lens 
type, focal length at acquisition, aperture range, aperture at acquisition. 

In addition, some parameter, in particular on the lens and the camera may be 
5 also stored in the device, which is the acquisition device such as the digital camera 
or the processing device such as the personal computer or digital printer. Such 
information may include parameters such as exit pupil, exit pupil distance regarding 
the lens, or distance of dust to sensor for the camera. 

10 A table with such data may look like: 



Field 


Example of data 


Category 


Lens Manufacturer 


Nikon 


lens 


Lens Type 


AF 24mm-45mm f2.8-f3.5 


lens 


Focal Length 


38mm 


Acquisition data 


Aperture 


f-16 


Acquisition data 


Dust distance 


0.156mm 


Camera data 


Exit pupil 


19mm 


Lens data 


Exit pupil distance 


230mm 


Lens data 



Table 2: Extracted Lens Parameters 



The dust map may also include meta-data that are different the list of 
15 extracted parameters. Moreover, that which is described as being extracted 

parameter dependent or encoded with extracted parameter value data or based on a 
value of an extracted parameter can be broadened to include other meta-data than 
just the extracted parameters listed in Table 2. For example, certain meta-data are 
dependent on parameters existing at the time of acquisition of the image, and can be 
20 camera-specific or not. The amount of ambient light available will depend on 

whether there is artificial lighting nearby or whether it is a cloudy day. Discussion of 
meta-data as it relates to image acquisition is found in more detail at U.S. patent 
application serial number 10/608,810, filed June 26, 2003, and is hereby 
incorporated by reference. 
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In the case that the system deals with multiple images (as defined in 102, 
103, and 104), the algorithm describes a loop operation on all images (110). The 
first step is to open a dust map (130). If non exists (132) the system will create a new 
dust map (200) as further described in Figure 2. In the case that the system has a 
5 few dust maps (1 30) the software will try to correlate one of the maps (300) to the 
image. This process is further illustrated in Figure 3. In particular this correlation 
refers to the adjustment of the shooting conditions to some accepted standard. The 
acquisition conditions in particular refer to the aperture and the focal length. The 
correlation process is interchangeable and can be done by adjusting the image to 

10 the map or adjusting the map to the image. In some cases both acquired image and 
dust map should be adjusted to some common ground. Such an example may 
happen when the map is calculated based on a aperture that the lens does not reach 
or a different lens than the one used wit different optical configuration. Alternatively, 
such as in the case of a new lens, this process (300) as called by 130, may be used 

15 to adjust the map onto a new map and from that stage onwards continue with a 
single map. 

If no dust map corresponds with the image (140), a new dust map is created 
(200). When a dust map does correspond (140) to the image, the preferred 
embodiment checks if the dust specs as defined in the dust map are of high enough 

20 confidence level to being dust regions (150). The statistical decision as to the way 
such confidence level is calculated for the dust map in general and for the individual 
dust specs, is further discussed in Figure 2 and Figure 3. If the confidence level is 
low, the image is added to the updating of the dust map (200). If after the image is 
added, the confidence level is high enough (152) the software continues to the dust 

25 removal process (160). Otherwise, the software progresses to the next image (170). 

For example, a dust map is considered valid after 10 images are analyzed, 
and a dust spec is considered valid after a dust is detected in 8 images. In this case, 
after analyzing 9 images, the software may continue to the stage of updating the 
dust map (200) but upon completion (10 images) there is a valid dust map, and the 

30 software will continue to the correction (160) stage. If however the loop (110) is only 
on its 1 st to 8 th image, no correction will be done. 

As an additional embodiment, images can be corrected retroactively after the 
dust map reached high enough confidence. This can be used for batch processing or 
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off line processing or processing where the information is gathered in parallel to the 
needed correction. In this embodiment, when the confidence level is not enough 
(152, NOT-YET) the images, a pointer to them, or a list of them, or a representation 
of them, that were used to create the dust map are stored in temporary location 
5 (154) and when the dust map is ready (151 -YES), the software will retroactively 
remove the dust from all those images (1 56). In this fashion, all images, including 
ones that originally did not hold sufficient statistical information, may be corrected. 

Referring to dust detection and correction (160), this process may be 
executed as a loop on every dust spec in the dust map, with a stage of detection and 

10 correction (400, followed by 166 and 168). Alternatively, the process can be 

implemented where all dust specs are detected first (162) and then all dust specs 
are corrected (164). The decision as to the sequence of operations varies based on 
implementation criteria such as what regions of the image are in memory, and 
should not affect the nature of the preferred embodiment. As part of the detection, 

15 the software also performs a self testing (500) where each new image introduced is 
compared to the dust map. This process is further defined in Figure 5. The 
importance of this stage for each dust spec and for the dust may, is that in this 
manner, if the dust situation changes, such as a single spec moving aground or the 
camera being serviced), the software will immediately detect the change and re- 

20 validate the dust map. As described above, the validity test (500) can be 
implemented on a dust spec by dust spec or on the full dust map. 

Referring to Figure 2-a, dust map creation and updating is illustrated. This 
process can receive a collection of images as defined by Figure 1 . blocks 108 and 
109) or one image at a time is refereed to this process, as defined by Figure 1 . block 

25 110. 

If the function is called with a single image (220-SINGLE IMAGE), the image 
is directly provided to the calculations (270). If multiple images are provided (240 
MULTIPLE IMAGES), then an initial step is to define if there are more than enough 
images for defining the map. This step is designed to optimize the creation process 
30 for dust in cases of a large amount of images. 

The sequence of the images are to be referenced based on the original 
collection of N images as defined in Figure 1 blocks 102, 103 or 104. The sequence 
of images is based on a few criteria such as: giving more weight to images shot last, 
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and if images are shot in a relatively small time frame, allocation in the sequence to 
movements of objects over large distances to try and assure minimal repetitiveness 
between similar images that may have been taken of the same object with little 
movement. The sequence will not be limited to the number of images (HIDP) 
5 because it may well be that some regions will not have enough data in them to 

evaluate the dust. This may happen in cases where part of the image is very dark in 
some of the images. 

As an example: if N (number of images in a selection) = 30; and HSdp 
(number of images needed to determining map) = 10; and all images were shot in a 
10 space of an hour; then a potential sequence may be: 



30,27,24,21,18,15,12,9,6,3,29,26,25 ,2,28,25, 1 



Alternatively if the same 30 images were taken over a period of a month it 
15 may be beneficial to select images sequentially (last one shot is the first to be 
calculated): 



30,29,28, 20,19 2,1 



20 And in some cases this process of sampling the series (270) may also decide 

not to use images that are taken too long from the last image. In this case, for 
example if image 1-15 were taken in July and 16-30 were taken in November, this 
process may limit the new map to 16-30 or even create two different maps, one for 
images 1 -1 5 and the other for 1 6-30. 

25 In a different criteria, the parameters as extracted from the images will 

determine the sequence and the number of dust maps that are to be calculated. For 
example if a folder contains N=30 images, where 15 were taken with one camera 
and 15 with another, the sampling step (270) may create two map sets. 

Another criteria for creating a new set or checking for new dust is the type of 

30 lens. If a lens is changed, it means that the CCD-cavity was exposed to potential 
new dust. This may trigger a new set of images to be looked at. It may also be an 
indication that the camera was serviced, or that the photographer cleaned the 
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camera. Of course, if there is a parameter that defines when a camera was serviced, 
this will trigger the creation of a new dust map. 

Those familiar in the art may be able to determine the right combinations of 
creating this sequence based on the nature of the dust, the camera and the lens. 
5 The next loop (270-271 ) defines the marking of each region and the addition of the 
region to the dust map if it is not already there. There are three type of regions in an 
image, the first are images with sufficient information to detect whether they are of 
the nature of dust, e.g., dark regions surrounded by a light background. Other criteria 
may be regions with a relatively small color saturation. The second group are regions 

10 that are very probably non-dust. Such regions are for example all clear, or of high 
color saturation. Other regions are inconclusive such as a very dark segment of the 
image. In this case, it will be hard to detect the dust even if it was part of the image. 
Alternatively, when looking for over-exposed or "dead pixels", the criteria may be 
reversed, if the pixels appear as a white spec in the image. 

15 The criteria may be also a function of the acquisition parameter. For example 

an image with a open aperture may all be marked as in-decisive, because the dust 
may not appear on the image. 

Regions that are potentially dust are marked (292) and then added to the dust 
mask (294). The addition may be the creation of a new dust spec on the map or the 

20 modification of the probability function or the confidence level counter for the region. 
Regions that are most likely non-dust are marked (282) and then added to the dust 
mask (284). The addition may be the creation of a new dust spec on the map or the 
modification of the probability function or the confidence level counter for the region. 
The additions of the regions needs to be normalized to the shooting conditions as 

25 defined by the dust map (300) if this step was not performed prior to entering this 
function, as optionally defined in Figure 1 . 

This loop continues over all regions of the image (271). Alternatively (272), 
each region is compared (500) with the map dust to se if there is no case where the 
monotonicity is broken, i.e. a region that was of high probability to be dust is now 

30 non-dust. 

Figure 2-b describes an alternate embodiment of a dust creation process. 
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Block 1210 describes the preparation of the dust map which includes either 
opening an existing map 1216 or creating a new one 1214, where all pixels as a 
starting point are non dust or WHITE. 

After the map is correlated to the shooting conditions, 300, a dust map l-dM is 
5 created in block 1220. 

All pixels in the image, 1222, receive the following values 1224 based on the 
luminance value of the pixel: 

If luminance is less than DARK THRESHLOD, then l-pM (X,Y) = 
MAYBEDUST; 

10 If luminance is greater than WHITE_THRESHLOD, then l-pM(x,y) = WHITE 

PIXEL; 

OTHERWISE l-pM(x,y) = DONT_KNOW. 



Once all pixels are analyzed, 1220, they are then clustered, 1296 into dust 

15 regions or dust specs in a 1-pM dust map. 

The next step is to create a dust map dM which is continuous based on the 
value of the individual pixels. In the final stage 1250, the dust map is threshold by 
the predetermined value THRESHOLD to create a binary mast. 

Figure 3 illustrates the process of correlating the image to a default settings of 

20 the dust map. This process defines correlating the image of the dust map, the dust 
map to a new dust map of the dust map to the mage. In particular this correlation 
refers to the adjustment of the shooting conditions to some accepted standard. The 
acquisition conditions in particular refer to the aperture and the focal length. The 
correlation process is interchangeable and can be done by adjusting the image to 

25 the map or adjusting the map to the image. In some cases both acquired image and 
dust map may be adjusted to some common ground. Such an example may happen 
when the ma is calculated based on a aperture that the lens does not reach or a 
different lens than the one used with a different optical configuration. Alternatively, in 
case of a new lens, this process (300) may be used to adjust the map onto a new 

30 map and from that stage onwards continue with a single map. 

To begin with, the dust map is being loaded (112) and the default data on 
which the map was generated is extracted (310). Such data may include the lens 
type, the aperture and the focal length associated wit the default state. In 
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concurrence, the information form the acquired image (304) is extracted (320) and 
compared to the one of the dust map. 

As explained in the mathematical model of the optical system, the two main 
adjustments between the dust map and the image are based on focal length, and on 
5 aperture, each creating a different artifact that should be addressed. Knowledge of 
the phenomena may assist in creating a better detection and correction of the dust 
artifact. Alternatively, in a separate embodiment, analysis of the image and the 
modification of the dust as changed by aperture and focal length, may provide the 
mathematical model that describes a transformation that defines changes to the dust 

10 as a function of change in lens type, focal length and aperture. 

The mathematical distortion of the dust as a function of the aperture is 
illustrated in figure 7a-7f. The geometrical optics illustrations of the above 
phenomena are depicted in Figures 6a-6e. Referring to figure 3, after extracting the 
data, the following step is modification of the map and or the image based no focal 

15 length (900), and based on aperture (800). The following steps are further defined in 
Figure 9 and Figure 8 respectively. 

Following the steps (800 and 900) the Image and the Dust Map are 
considered to be correlated. The correlated map cM is no longer binary because it 
defines both the shift and the fall off which is continuous. Figure 400 defines the 

20 process of detecting and removing the dust from the image. The input is the image I 
is loaded, if it is not already in memory (404) and the correlated dust map is cM is 
loaded (402) if not already in memory. 

The process of detecting and removing the dust is done per dust spec. This 
process is highly parallelized and can be performed as a single path over the image, 

25 or in strips. The flexibility of performing the operation on the entire image, or in 

portions, combined with the correlation or as a separate process, enables a flexible 
implementation of the algorithm based on external restrictions defined by the 
hardware, the run time environment, memory restrictions and processing speed. 

As defined and justified by physical phenomena, a method of correcting dust 

30 artifacts is described based on two different operations. The former is the retouching 
or the in-painting here regions with no data (420), or data that is close to noise to be 
recreated as defined in 430 and later on in Figure 7e, 7f. The second portion of the 
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correction is based on the tapering degradation of surrounding pixels as a function of 
the aperture, as illustrated in Figures 7b, 7c, 7d. 

Referring to the image enhancement portions, where the data exists but most 
likely not in its full form, due to the fact that some of the dust affects the quality of the 
5 image, but some regions still contain some data (420), an image enhancement 
algorithm is performed on the pixels (430). In a simplified embodiment (432), 
assuming that the optical model is simplified to a degradation in the overall 
brightness as defined by the OPACITY, the enhanced pixels will receive a value 
inversely related to the OPACITY i.e. 9432) 

10 

z'(*,jO= I(x ' y) 

OPACITY 

To maintain a check of the validity of the model, the pixels before being 
operated on may be provided for validation (500) as described in Figure 5. The 

15 second portion of the image correction is the restoration or the inpainting (460). In 
this case, the area behind the dust has no relevant data to enhance, or if there is, 
this data is relatively close to the overall noise level and thus can not be enhanced. 
Therefore, there is a desire to in-paint every pixel based on the analysis of the 
surrounding region to the dust (470). In the case where the enhancement regions as 

20 defined in block 430 are of good quality, those pixels as well may contribute to the in- 
paining. Otherwise, they are excluded form the calculations. The decision may rely 
on the distance of the pixels to the OPAQUE ones, of the minimum level of 
OPACITY that was used to restore the pixels in block 430. The larger the OPACITY, 
the smaller such pixels may be relied on in the inpainting process. 

25 Figure 4b illustrates an embodiment of the in-painting process. In general, 

each pixel in the obscured region, 480 is to be filled, 482, based on its surrounding 
pixels. In this specific embodiment, the pixels are filled-in based on information of 
the surrounding non-affected pixels. This specific algorithm takes into account that 
the pixels closest to the periphery have a better chance to be anticipated by the 

30 external pixels. Therefore the process is done from the outside inwards. 

This process of spiraling inwards is also depicted in Figure 4d. In it, given a 
dust spec 1400 in a grid 1410, the dust takes a digital form of the surrounding 
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bounding area 1420. External pixels such as 1430 are not part of the dust region, 
while internal ones such as 1450 are. 

An algorithm that works on the periphery moving inwards is described with 
reference to Figure 4b. Block 470 is depicted in Figure 4d as follows: In the first 
5 round all peripheral pixels, numbered form 1 to 20 are being operated on as defined 
in Figure 4b, block 474. After that, all of the above mentioned twenty pixels, as 
defined in Figure 4D block 1460 are removed, according to block 476 of Figure 4b, 
and a region is formed leaving a smaller dust spec of shape 1470. This modified 
dust spec has a different set of peripheral pixels numbered 21-31 . After removing 

10 those pixels by the in painting process, block 467, a smaller dust kernel as depicted 
in 1480 is left with only three pixels 34, 35 and 36. 

The process of filling in the pixels need not be spiral. In a different alternative, 
the process follows the pattern of the region surrounding the dust. For example lines 
and edges, or other high frequency information are being filled in prior to filling in the 

15 smooth regions. This approach will prevent unnecessary blurring due to the in- 
painting process. 

In a preferred embodiment, a criteria for the chosen value can be based on 
marinating the overall gradient around the pixel 462. The justification is that in the 
case of a steep edge, it is not advisable to copy information for the surrounding 

20 pixels with the edge value. By doing so, the in painting process will maintain any high 
frequency information in the picture such as lines and edges. 

An illustration of that is given in figures 4c, 4d and 4e. Referring to figure 4c, 
the same dust 1400 as in figure 4b with the bounding box 1420 on a grid 1410 is 
obscuring a picture including high frequency data such as the letter A, 1440, by 

25 taking a small section of based on 9 pixels as depicted in figure 4d pixel 1490 is 
surrounding by 8 pixels 1491, 1492, 1493, 1494, 1495, 1496, 1497, 1498. 

In a simplification, each pixel receives the average value between the black 
and white regions. This is depicted in figure 4e. In this case, the pixel 1490 and the 
surrounding 8 pixels 1491, 1492, 1493, 1494, 1495, 1496, 1497, and 1498, have a 

30 digital value of 21 , 220, 250, 245, 253, 1 80, 1 45 and 35 respectively. It is clear that 
the best value of pixel 1490 will come form its surrounding pixels with the smallest 
gradient. Because in practice the value of pixel 1490 can not be determined, but is 
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the center of the in painting, the value of the gradient will be based on the value of 
the gradients of the pixels around it and extrapolated to this pixel. 

In this specific example, the differential horizontal gradient between pixels 
1497 and 1493 is the largest while the gradient vertical gradient of 1495 and 1491 
5 will most likely be the same. Therefore, a preferred value of 210 will be based on the 
extrapolated average of the two gradients of its top and bottom pixels. 

Figure 500 describes a tool that may be used in various stages of the 
process, namely quality control. This process can be called from the main workflow 
(Figure 1 ) the creation of the dust map (figure 3) or the image restoration and 
10 enhancement (Figure 4 and 4-a respectively). 

A quality check may be performed to assure that pixels will not be wrongly 
classified, nor mistakenly corrected when they should not. 

The quality control test can be applied to a pixel in an image which belongs to 
a potential dust (510). In case the image has not yet been correlated with the dust 
15 (300), this process should then be done. The next step is to check whether the pixel 
belongs to a dust region or not. For example, in a small aperture, the region behind 
the dust should be close to if not totally obscured. If not (530 NO), there is a 
remaining desire to revive the dust map (550) or the image does not correspond to 
the dust map. This can happen when the image is from a different time, or from a 
20 different acquisition device. In this case, the software will create a new dust map 
(200) to correspond to the new dust or lack thereof, or inform the user that it cannot 
correct this image. In case where there is not enough information (530 MAYBE), 
there is no conclusive evidence to reject the correlation and thus the process 
continues (580) with no action. In the case that the image displays information that 
25 may concur with the dust map, the software may continue (580) or prior to that, 
enhance the likelihood in the dust map (200) that the region is indeed dust. 

Figure 8 describes the adjustment of the dust map to the image acquisition 
parameters based on the aperture. Simplifying, the closer aperture is, the crisper 
and more noticeable the dust is. In other words, for example, for images taken with 
30 an f-stop of f-32, the dust will be very prominent and opaque, while the same image 
taken at f-2.8 may display no visual degradation of the image. 

The correction of the image should take that information into account, to 
prevent over-correction of the dust artifact. 
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The acquisition information and the corresponding dust map default setup are 
extracted in blocks 326 and 312, respectively. Then, for each dust spec in the image 
810, the size of the region that is still obscured by the dust is calculated, as defined 
by mathematical model. In some cases, when the aperture is very open, this region 
5 may decline to 0. In others, where the aperture is still very close, the size may be 
close to the size of the dust. Alternatively, this step, 820, may be done as a 
preparation stage, and kept in a database, which can be loaded. 

The process then splits in two. The fully obscured region will be marked in 
834 pixel by pixel 832 in a loop 834, 835 and will be treated by the in-painting 

10 process as defined in Figure 4-a. A semi-opaque dust map is created in the loop 
840, 841 for each pixel. Each of the pixels 842 is assigned an OPACITY value 844, 
based on the mathematical model as described previously in Figure 7a-4d. The dust 
spec that is only partially attenuated will go through an inverse filtering of the 
information already there, as described in Figure 4 block 430, with a specific 

15 embodiment in block 432. The process of the inverse filtering may take into account 
the signal to noise ratio to avoid enhancing data which is not part of the original 
image. For example, the region around the dust may have an over-shoot similar to a 
high pass filter, which may manifest itself in the form of an aura around the dust. This 
aura should to be taken into account before enhancing the regions. 

20 Figure 9 describes the adjustment of the Dust Map based on the Focal length, 

and the specific lens. The scientific background is explained in figures 7e-7f. As 
described before, the shift of the dust spec as a function of focal length for a specific 
lens is a function of equation 1 3 the thickness of the window- tw, which is constant 
for a given camera and exit pupil position- Pe which varies based on the lens system 

25 and the variable focal length in case of a zoom lens. 

Given an image and a dust map, the pertinent information is loaded, as 
described in Figure 3, namely the focal lens and lens type of the camera, 326, the 
focal length and lens type in the dust map 312 and the camera distance of dust to 
the sensor 318. 

30 The process then goes through the dust specs in the image 910 and 

calculates the shift of the dust as explained in figure 7a-7e. The coordinates of the 
pixel are calculated from the center of the optical path, 922, and the shift is 
calculated 924. Alternatively to going through each pixel, in order to speed the 
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process, only the periphery of the dust spec can be calculated and the rest will be 
filled in. Moreover, because the shift is a function of the location (x,y) of the dust, in 
the case where dust is far enough from the origin, the dust shape will not change. It 
is then sufficient to calculate only a shift of a single point and displace the entire dust 
5 spec accordingly. The regions, which are only partially attenuated due to the change 
in aperture, may be calculated in this stage 940, 941 , if calculated already as 
illustrated in figure 8, or alternatively, the displacement can be calculated first as 
explained in blocks 942,944. 

In some cases, it is impossible to get data on the exit pupil, or the distance the 

10 dust is from the sensor. Such cases may be when the application has no a-priori 
knowledge of the camera or the lens that was used. 

It is still possible in those cases to create a reasonable dust map, by empirical 
reconstruction of the parameter based on analysis of the images, the shift in the dust 
and the falloff of the dust edges. Such techniques are defined in Figures 1 0 and 1 1 

15 illustrating a process of estimating, based on empirical data, the values of the 

parameters in the formulae that defines the change in dust as a function of change in 
aperture. 

This process can be useful in the case that the software has no a-priori 
knowledge of the extracted parameters of the camera or lens that are used to 
20 convert the acquired image and the dust map image into the same known space. In 
an alternative embodiment, this technique can also be used to extract optical 
parameters, similar to methods done in a controlled laboratory settings such as using 
an optical bench of a lens regardless of the desire to remove dust artifact from an 
image. 

25 Alternatively, this process can be used with partial knowledge, such as the 

shift of a single dust spec to evaluate the transposition of all dust specs in the image. 

Figure 10 defines the process of creating the mathematical formulae based 
on empirical data to parameterize the change in dust as a function of change in focal 
length. 

30 In general, a proposed embodiment relies on the fact that when dust is found, 

a pattern matching can be applied to find the shift in the dust. Based on this 



WO 2007/095483 



PCT/US2007/061964 



-30- 

information, ~ can be calculated as defined in Equation 13-a. If l w is known then 

e 

Pe can be calculated as recited in equation 1-b. 

Specifically, in a preferred embodiment, an image is acquired, 1010 and a 
dust map is calculated 1012. A second image is captured, 1030 with a different focal 
5 length than the first image, and a dust map is calculated 1032. The process 

repeatedly tries to find two dust spots in the two dust maps 1040. If no dust specs 
are correlated, the process is repeated for consecutive images 1020. 

The process of finding dust specs is calculated by applying a local correlation 
to each of the dust specs. Preferably, based on Equation 13-a, the further the dust is 
10 from the center, the better the precision is. 

When two specs are determined to be from the same dust spec, the disparity 
between the specs is calculated 1042. The ratio between the shifted pixels is 

calculated. This ratio is the numerical empirical estimation of K in equation 13-a. 
Moreover, if the distance of the dust to the sensor is known, the exit pupil of the lens 
15 can be calculated based on the same equation. 

Figure 11 illustrates a process of estimating based on empirical data the 
values of the parameters in the formulae that define the change in dust as a function 
of change in aperture. 

The process is similar to the one described for estimation of the focal length, 
20 albeit the parameters calculated are different. 

Specifically, a first image is acquired, 1110 and dust is detected, 1 120 in this 
image. If the image is not appropriate for detecting the dust, or if the probability is 
low for the dust regions, this image is rejected for this purpose and another image is 
chosen. For the empirical comparison, a second image is captured 1 140, or 

25 alternatively a set of images 1 130, all with varying aperture, to enable acceptable 
sampling set. The process then looks for a detected dust region with high level of 
probability. In the case that all other parameters are similar except the aperture, the 
process can search for dust regions in the same coordinates that the original image 
dust regions were found. The dust regions of the two or more images are correlated 

30 1 1 60. The process continues for a sufficient amount of dust regions, 1 1 68, which in 



WO 2007/095483 



PCT/US2007/061964 



-31 - 

some cases can even be a single one, and sufficient amount of images, 11 69, which 
can also be, depending on the confidence level, a single image. Once the dust 
regions are correlated 1 160, the fall off due to the change of aperture is calculated, 
1 1 72, on a pixel by pixel basis, 1 1 70, for every pixel in a dust region, 1 1 79. Based 
5 on this information, the fall-off function is calculated. 1 180. In a preferred 

embodiment, the fall-off function is determined as a function of the distance of a pixel 
from the periphery of the dust, as well as the aperture. 

Alternatively, the dust specs may also be determined by trying to correlate the 
dust spec in the map to the acquired image. Such correlation can be performed 
10 within a reasonable window size or even on the diagonal line between the dust and 
the center of the optical path, in the direction that the dust may move. By gaining 
knowledge on the movement of a single dust spec, as explained in formulae 13a- 
13d, all other dust specs shift can be determined. 

It is also possible to determine whether the camera should be physically 

15 cleaned based on analysis of the dust in the camera and the specific dust patterns. 
This is illustrated in the flowchart of figure 12. An example will be that a certain width 
of dust will not allow correct in-painting based on the surroundings. Another example 
will be the overall number of dust specs or the overall relative area that the dust 
covers. The input for such analysis is a dust map, 1 200. This map can be similar to 

20 the dust map generated in block 200, or any other representation of a dust map or a 
calibration map, generated automatically or manually by the photographer. Such 
analysis need not be performed for every image. A process, 1210 determines 
whether to perform such analysis. Examples to trigger this process are the time 
since the last analysis, change of lenses, which may create the introduction of dust, 

25 or message from the quality control of dust map, as defined in Figure 5 block 500, 
that the dust has shifted or that the dust no longer corresponds to the map. In 
general, any changes in the dust structure may be a justification to trigger the 
analysis process. When no analysis is desired, 1212, the process terminates. 
Otherwise, the analysis is performed, 1220. The analysis is performed for each dust 

30 spec individually, 1 130, and then the results are accumulated for the whole image. 
For each dust region, 1 140, some parameters are being extracted, including, but not 
limited to: area of dust region, maximum width of dust region; distance between dust 
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region and neighboring regions; movement of dust region from last analysis; and 
occurrence of new dust specs since last analysis. Following this analysis, the results 
are summarized to include, e.g., overall area of dust, in size and in percentage for 
the entire image, largest width of dust, largest area of dust spec, changes in area of 
5 dust since last analysis, and changes of dust particles since last analysis. In an 
alternate embodiment, either automatically or based on the photographers 
preference, this analysis may be displayed or saved in a log file for future reference 
1221 . Each of the aforementioned criteria may have an acceptability threshold value. 
This threshold value is determined empirically by the manufacturer in terms of the 

10 maximum acceptable dust that can be corrected in software. Alternatively, this data 
may be adjusted by the photographer based on her tolerance level to dust. If any of 
the parameters exceeds an acceptable threshold, the user is informed, 1290, that 
the camera should be manually maintained and cleaned up. 

Alternatively, 1285, this same process may be used as a tool to inform the 

15 user of changes in the dust. Such information is particularly important in the case 
that the dust correction algorithm is based on the creation of a calibration image. In 
this case, the analysis will be used to inform the user that a new calibration image 
should be acquired to support the correction of dust in future images. 

Alternatively, the process of analysis described above may also be 

20 incorporated in a maintenance procedure, where the camera, after being cleaned up 
will perform an auto-test, as described in Figure 12, to verify that the camera now is 
indeed clean. In such cases, the threshold parameters are substantially more 
restrictive and demanding, to assure high quality maintenance. 

There are many alternatives to the preferred embodiments described above 

25 that may be incorporated into a image processing method, a digital camera, and/or 
an image processing system including a digital camera and an external image 
processing device that may be advantageous. For example, an electronic circuit 
may be designed to detect maximum and minimum dust detection probability 
thresholds while acquiring pixels in an image (see also US patent 5,065,257 to 

30 Yamada, hereby incorporated by reference). Such a circuit can produce signals that 
may be used in processing to eliminate regions, which lie outside the expected range 
of signal values due to the presence of dust particles or similar optical defects, or 
alternatively to accept, maintain or eliminate regions as dust artifact regions based 
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on whether a probability determination exceeds, high or low, a certain threshold or 
thresholds. A technique may be used to detect and provide a remedy for the effects 
of dust on a digitally-acquired image (see also US patent 5,214,470 to Denber, 
hereby incorporated by reference). An image may be recorded within a digital 
5 camera or external processing device such as may be signal coupled with a digital 
camera for receiving digital output image information or a device that acquires or 
captures a digital image of a film image. The digital image may be compared with 
the film image, e.g., through a logical XOR operation, which may be used to remove 
dust spots or blemishes picked up in the acquisition of the digital image from the film 
10 image. 

Multiple images may be processed and stationary components, which are 
common between images, may be detected and assigned a high probability of being 
a defect (see also US patent 6,035,072 to Read, hereby incorporated by reference). 
Additional techniques, which may be employed to modify defect probability, may 

15 include median filtering, sample area detection and dynamic adjustment of scores. 
This dynamic defect detection process allows defect compensation, defect correction 
and alerting an operator of the likelihood of defects. 

Dark field imaging may be employed to determine the location of defects in 
digital images from digital cameras or film scanners (see US patent 5,969,372 to 

20 Stavely et al., and US patent application 2001/0035491 to Ochiai et al., each hereby 
incorporated by reference). A normal imaging of a object with normal illumination 
may be followed by a second imaging using different wavelengths, e.g., infrared 
illumination. Dust, fingerprints, scratches and other optical defects are typically 
opaque to infrared light. Thus the second image produces an image with dark spots 

25 indicating the position of dust particles or other defects. 

A process may involve changing any of a variety of extracted parameters (see 
elsewhere herein), angle of sensor relative to image plane, distance of image plane 
or sensor from dust specks (e.g., on window of sensor), etc., and imaging a same 
object with the digital camera. A comparison of the images reveals with enhanced 

30 probability the locations of dust artifact. In a camera application, the unique location 
of the actual dust relative to the object and to the image plane provide information 
about extracted parameter-dependent characteristics of dust artifact in the images. 
The analysis for the digital camera application depends on the "transmission"-based 



WO 2007/095483 



PCT/US2007/061964 



-34 - 

optical parameters of the system, i.e., the fact that light travels from a scene through 
the camera lens and onto the camera sensor, and not involving any substantial 
reflective effects. It is possible to make determinations as to where the dust actually 
is in the system by analyzing multiple images taken with different extracted 
5 parameters, e.g., on the sensor window, or in an image of an original object which 
itself is being images such as in film imaging. 

In a scanning application, this technique can be use the face that a speck of 
dust will cast a shadow of a different color, geometry location, etc. with changes in 
extracted parameters, e.g., with a different color with increasing elongation of the 

10 shadow for each parallel row of pixels (a "rainbow" shadow, as it were). Multiple 
scans taken from various angles of illumination may be employed to produce an 
image which identifies dust defects from their shadows and the colors thereof (see 
US patent 6,465,801 to Gann et al. and US patent applications 2002/0195577 and 
2002/0158192 to Gann et al, hereby incorporated by reference). A linear scanning 

15 element moves across a document (or the document is moved across the scanning 
element) and an image of the document is built up as a series of rows of pixels. This 
differs from the physical configuration of a camera in which a shutter illuminates a X- 
Y sensor array with a single burst of light. In both cases, though, dust may lie close 
to the imaging plane of the sensor. 

20 Technique may be applied as part of a photofinishing process to eliminate 

blemishes on a film image obtained by a digital camera (see also US patent 
application 2001/0041018 to Sonoda, hereby incorporated by reference). Such 
techniques may import previous acquired information about defects in images from a 
blemish detection procedure. A technique for correcting image defects from a digital 

25 image acquisition device such as a digital camera may involve repeated imaging of 
an object or other image, where each successive image-acquisition involves different 
properties or extracted parameters or meta-data related properties, such as variable 
angles of incidence or variable lighting or contrast parameters, and the results of 
these repeated scans may be combined to form a reference image from which defect 

30 corrections are made (see also US patent application 2003/01 18249 to Edgar, 
hereby incorporated by reference). A decision on whether a defect in an image 
acquired by a field-based digital camera is to be corrected or not may be based on a 
balancing of considerations. For example, the likely damage to surrounding defect- 
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free portions of the image may be balanced against the likelihood of successfully 
achieving correction of the defect. 

Image processing means may be employed where the detection or correction 
of defects in a digital image may be based solely on analysis of the digital image, or 
5 may employ techniques directly related to the image acquisition process, or both. 
Anomalous image regions may be determined based on the difference between the 
gradient of an image at a set of grid points and the local mean of the image gradient 
(e.g., see US patent 6,233,364 to Krainiouk et al., hereby incorporated by reference). 
Such technique can reduce the number of false positives in "noisy" regions of an 

10 image such as those representing leaves in a tree, or pebbles on a beach. After 
determining an initial defect list by this means, the technique may involve culling the 
list based on a one or more or a series of heuristic measures based on color, size, 
shape and/or visibility measures where these are designed to indicate how much an 
anomalous region resembles a dust fragment or a scratch. 

15 Techniques and means to correct scratches in a digitized images may employ 

a binary mask to indicate regions requiring repair or noise removal, and sample and 
repair windows to indicate (i) the region requiring repair and/or (ii) a similar "sample" 
area of the image (see also US patent 5,974,194 to Hirani et al., hereby incorporated 
by reference). Data from a sample window may be converted to a frequency domain 

20 and combined with frequency domain data of the repair window. When a low-pass 
filter is applied, it has the effect to remove the sharp, or high-frequency, scratch 
defect. 

Techniques and means of detecting potential defect or "trash" regions within 
an image may be based on a comparison of the quadratic differential value of a pixel 

25 with a pre-determined threshold value (see US patent 6,125,213 to Morimoto, hereby 
incorporated by reference). The technique may involve correcting "trash" regions 
within an image by successively interpolating from the outside of the "trash" region to 
the inside of this region. 

Techniques and means to automate the removal of narrow elongated 

30 distortions from a digital image may utilize the characteristics of image regions 
bordering the distortion (see also US patent 6,266,054 to Lawton et al., hereby 
incorporated by reference). User input may be used to mark the region of the defect, 
or automatic defect detection may be employed according to a preferred 
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embodiment herein, while the process of delineating the defect is also preferably 
also performed automatically. 

Techniques and means to allow automatic alteration of defects in digital 
images may be based upon a defect channel having a signal proportional to defects 
5 in the digital image (see also US patent 6,487,321 to Edgar et al., hereby 

incorporated by reference). This allows areas of strong defect to be more easily 
excised without causing significant damage to the area of the image surrounding the 
defect. 

Techniques and means may be employed to generate replacement data 
10 values for an image region (see also US patent 6,587,592 to Georgiev et al., hereby 
incorporated by reference). Image defect may be repaired as facilitated by the 
replacement data. Moreover, the repairing of the unwanted image region may 
preserve image textures within the repaired (or "healed") region of the image. 

Techniques and means may be employed to detect defect pixels by applying 
15 a median filter to an image and subtracting the result from the original image to 

obtain a difference image (see also US patent application 2003/0039402 and WIPO 
patent application WO-03/0 19473, both to Robins et al., each hereby incorporated by 
reference). This may be used to construct at least one defect map. Correction of 
suspected defect pixels may be achieved by replacing those pixel values with pixel 
20 values from the filtered image and applying a smoothing operation. User input may 
or may not be utilized to further mitigate the effects of uncertainty in defect 
identification. 

Techniques and means for retouching binary image data which is to be 
presented on a view-screen or display apparatus may be employed to eliminate local 

25 screen defects such as dust and scratch artifacts (see also US patent application 
2002/0154831 to Hansen et al., hereby incorporated by reference). The production 
of visible moire effects in the retouched image data may be avoided by the 
replacement of small areas. 

A digital video camera with sensor apparatus may incorporate a defect 

30 detecting mode (see also US patent 5,416,516 to Kameyama et al., hereby 

incorporated by reference). The locations of detected defect pixels may be retained 
in the memory of the camera apparatus and replacement pixel values may be 
interpolated by processing algorithms, which convert the sensor data into digital 
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image pixel values. Techniques may be employed to automatically detect and 
compensate for defective sensor pixels within a video camera (see also US patent 
5,625,413 to Katoh et al., hereby incorporated by reference). The camera may 
perform a dark current measurement on start-up when the camera iris is closed and 
5 by raising the gain can determine pixels which exhibit abnormally high dark current 
values. The location of these pixels is recorded in camera memory as a LUT with 
associated threshold brightness values associated with each pixel depending on its 
dark current value; defect compensation depends on input image brightness and 
ambient temperature. 

10 An image pickup apparatus, such as a digital camera, may have a detachable 

lens (see also US patent application 2003/0133027 to Itoh, hereby incorporated by 
reference). The camera may incorporate a defect detecting section and a 
compensation section for sensor defects. Further the defect detection operation may 
become active when the camera lens is detached so that the user will not miss an 

15 opportunity to take a picture due to the operation of the defect detection process. 

The techniques of the preferred and alternative embodiments described 
herein may be applied to printers and to imaging devices such as digital cameras 
which incorporate a focusing lens system. A process may be employed for detecting 
and mapping dust on the surface of a photographic element (see also US patent 

20 5,436,979 to Gray et al., hereby incorporated by reference). This may be applied in 
the context of a verification procedure to follow a cleaning process for a range of 
photographic elements including film negatives and slides. Statistical information 
may be obtained and presented to an operator to allow control of the cleaning 
process. Detailed location information may be also recorded and/or correction 

25 means may be also provided for dust defects on a photographic element. 

Techniques and means to create a defect map for a digital camera or similar 
imaging device may use an all-white reference background (see also US patent 
application 2002/0093577 to Kitawaki et al., hereby incorporated by reference). The 
location of any dust or scratch defects may be recorded in the memory of the 

30 imaging apparatus when the camera is in a dust detection mode and when a dust 
correction circuit is active any image data co-located with a defect may be corrected 
for the presence of dust by elimination, color correction or interpolation based on the 
surrounding pixels. Further, where the position of a dust defect changes with f-stop 
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of the camera a list of dust locations corresponding to f-stop settings is pre recorded 
at the time of manufacturing in a LUT in the camera memory. Any effect of different 
focal length may be simplified to the effect of the change in dust due to magnification 
of the lens. In addition, techniques for dynamically obtaining a defect map based on 
5 the processing of a plurality of images may be employed with this technique. 

Techniques may be also employed involving correcting for dust defects based 
on the geometry of said dust or of the camera. Further techniques may involve 
utilizing camera metadata to enhance the detection and correction processes for 
defects. Further techniques may involve alerting the user that the camera requires 

10 servicing due to excessive levels of dust contamination, or the fact that it is not only 
magnification but the actual lens that is mounted. 

A method of filtering dust artifacts form an acquired digital image including 
multiplicity of pixels indicative of dust, the pixels forming various shapes in the 
image, may be employed. The method may include analyzing image information 

15 including information describing conditions under which the image was acquired 
and/or acquisition device-specific information. One or more regions may be 
determined within the digital image suspected as including dust artifact. Based at 
least in part on said meta-data analysis, it may be determined whether the regions 
are actual dust artifact. 

20 A method may include obtaining a dust map based on analysis of multiple 

images acquired by the same image acquisition device. The dust map may include 
regions of various shapes and sizes indicative of statistically recurring patterns 
associated with dust. 

A method may further include analyzing the images in comparison to a 

25 predetermined dust map to establish the validity of the dust over progressions of 
time. The method may further involve mapping the acquired image to a 
predetermined default acquisition condition as a function of the lens type and the 
focal length that was used at acquisition. 

A method may further include mapping a dust spec as depicted in the dust 

30 map and the suspected dust specs in the acquired image based on a calculated 
transformation of the dust as a function of the lens and the aperture, or other 
extracted parameter, used to acquire the image. The actual removal of dust artifacts 
from an image may include a step where missing data as obscured by the dust 
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specs is regenerated and in-painted based on analysis of the region in the image 
surrounding the dust spec. The actual removal of the dust artifacts from the image 
may also include a step where deteriorated regions primarily in the periphery of the 
dust spec are enhanced and restored based on knowledge of the deterioration 
5 function. The actual image retouching may include both in-painting and restoration 
or either one of these operations, or another image correction technique as may be 
understood by those skilled in the art. 

A method of detecting and removing dust artifacts may be performed in the 
acquisition device as a post-processing stage prior to saving the image. This 

10 method may further include an analysis of the image in its raw format immediately 
followed by the acquisition stage. The method of detecting and removing dust 
artifacts can be performed on an external device as part of a download or capture 
process. Such external device may be a personal computer, a storage device, and 
archival device, a display or a printing device or other device. The method of 

15 detecting and removing dust artifacts can be performed in part in the acquisition 
device and the external device. 

A dust detection and/or correction technique may be applied post priori to a 
collection of images, or individually to images as they are added to a collection. The 
map may be generated a priori to the introduction of an image, or dynamically and in 

20 concurrence to the introduction of new images. The method may further include 
steps of providing a statistical confidence level as to the fact that a certain region is 
indeed part of a dust spec. The method may further provide tools to determine 
whether the acquisition device may benefit from some maintenance. 

A method may be employed that may be implemented as part of a digitization 

25 process, such as correcting defects on scanning device, whether flat bed or drum, 
whether for hard copy documents or for film digitization. A method may be further 
applied to other recurring image imperfections such as dead pixels on the sensor, 
burnt pixels on the sensor, scratches, etc. A method of automatically determining 
whether to recommend servicing a digital image acquisition system including a digital 

30 camera based on dust analysis may be advantageously employed. A method of 
calculating parameters of an optical system may be based on analysis of the dust. 

While an exemplary drawings and specific embodiments of the present 
invention have been described and illustrated, it is to be understood that that the 
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scope of the present invention is not to be limited to the particular embodiments 
discussed. Thus, the embodiments shall be regarded as illustrative rather than 
restrictive, and it should be understood that variations may be made in those 
embodiments by workers skilled in the arts without departing from the scope of the 
5 present invention as set forth in the claims that follow and their structural and 
functional equivalents. 

Particularly, any of the techniques described above or claimed below may be 
combined in alternative embodiments with any of the following. A method of 
automatically correcting dust artifact within images acquired by a system including a 

10 digital acquisition device including a lens assembly is provided. The method 

includes acquiring multiple original digital images with the digital acquisition device. 
Probabilities that certain pixels correspond to dust artifact regions within the images 
are determined based at least in part on a comparison of suspected dust artifact 
regions within two or more of the images. Probable dust artifact regions are 

15 associated with one or more values of one or more extracted parameters relating to 
the lens assembly of the digital acquisition device when the images were acquired. 
A statistical dust map is formed including mapped dust regions based on the dust 
artifact determining and associating. Pixels corresponding to correlated dust artifact 
regions are corrected within further digitally-acquired images based on the 

20 associated statistical dust map. 

The dust artifact probabilities may be determined by statistically combining 
multiple individual probabilities based on each of the regions within two or more 
images. Probabilities may alternatively be determined that certain pixels within the 
image correspond to regions free of dust based at least in part on a comparison of 

25 suspected dust artifact regions within one or more of images. Certain suspected 
dust artifact regions may be eliminated based on probabilities that certain pixels 
correspond to regions free of dust, and/or as having a determined probability below a 
first threshold value that the suspected region is in fact dust artifact. 

Certain further dust artifact regions may be judged as having a probability 

30 above the threshold value, such as to be subject to further probability determining 
including comparison with further acquired images prior to judging whether each 
further dust artifact region will be subject to the eliminating operation. There may be 
only a single threshold such that above the threshold, the suspected region will be 
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judged to be subject to a dust correction operation, or a second threshold above the 
first threshold may be applied for that purpose, and there may be further thresholds 
defined within the software. 

The probability determining operation may include weighting suspected dust 
5 artifact regions according to one or more predetermined probability weighting 
assessment conditions. These conditions may include size, shape, brightness or 
opacity, feathering, or peripheral smoothness of said suspected dust artifact regions, 
or a degree of similarity in size, shape, brightness, opacity or location with one or 
more suspected dust artifact regions in one or more other images, or combinations 
10 thereof. 

The one or more extracted parameters may include aperture size, F-number, 
magnification, lens type, focal length of an optical system of the digital camera, or 
combinations thereof. The one or more extracted parameters may be calculated 
empirically from comparison of one or more dust artifact regions within multiple 

15 original digital images with the digital acquisition device. 

The further digitally-acquired images may include different images than the 
originally acquired images, and may further include one or more of the same original 
images. The different images may be acquired with different values of one or more 
extracted parameters, they may be acquired of a different object or objects or of a 

20 different scene or scenes. 

The determining and associating operations may be repeated for the further 
digitally-acquired images, and the statistical dust map including mapped dust regions 
may be updated based on the additional dust artifact determining and associating. 
The determining and associating may also be repeated for additional images 

25 acquired with the digital camera, and the statistical dust map including the mapped 
dust regions may be updated based on the additional dust artifact determining and 
associating. Pixels may be corrected corresponding to correlated dust artifact 
regions within the additional images based on the updated, associated statistical 
dust map. One or more of the further and original images may be updated based on 

30 the updating of the associated statistical dust map. The updating for the further and 
original images may be limited to updates that do not include appearance of new 
dust or movement of existing dust, and/or to updates that include previously 
determined dust artifact regions. 
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A version description may be created of changes in the statistical dust map. 
One or more of the further and original images may be updated based on the 
updating of the associated statistical dust map based on the version description. 
The version may be based on a chronological time stamp, and/or replacement of a 
5 lens. Version information may include a change of probability in the statistical dust 
map that certain pixels correspond to dust artifact regions. The version information 
may include one or more parameters including a change in dust location, change in 
dust position, appearance of a new dust region, and/or a disappearance of an 
existing dust region. A determination may be made as to whether a dust map is to 

10 be replaced based on determining that sufficient disparity exists based on an amount 
and/or quality of changes in the statistical dust map. 

The image correction method may be automatically performed within a digital 
camera that includes an optical system, a sensor array, processing electronics and a 
memory. The image correction method may also be performed at least in part within 

15 an external processing device that couples with a digital camera that includes the 
optical system and the sensor array to form a digital image acquisition and 
processing system that also includes the processing electronics and the memory. 
The programming instructions may be stored on a memory within the external device 
which performs the image correction method. The digital acquisition device may 

20 capture images from film images, and the digital acquisition device may include a 
digital camera. 

The dust artifact probability determining may include dynamically updating the 
probabilities based on comparisons with suspected equivalent dust artifact regions 
within the further digitally-acquired images. The determining of probabilities is 

25 further based on a pixel analysis of the suspected dust artifact regions in view of 
predetermined characteristics indicative of the presence of a dust artifact region. 
The dust artifact probability determining may include determining probabilities that 
certain pixels correspond to dust artifact regions within acquired images based at 
least in part on a comparison of suspected dust artifact regions within two or more 

30 digitally-acquired images, or on a pixel analysis of the suspected dust artifact regions 
in view of predetermined characteristics indicative of the presence of a dust artifact 
region, or both. 



WO 2007/095483 



PCT/US2007/061964 



-43- 

The probability determining may further include statistically combining 
probabilities based on comparisons of inner or shadow regions and of aura regions. 
The probability determining with respect to a shadow region of a dust artifact may be 
based on an extracted parameter-dependent shadow region analysis, wherein the 
5 shadow region analysis presumes that certain regions on a sensor of the digital 

image acquisition device are fully obscured by the dust. The probability determining 
with respect to an aura region of the dust artifact may be based on an extracted 
parameter-dependent aura region analysis, wherein the aura region analysis 
presumes that certain regions on a of the digital image acquisition device are 

10 partially obscured by the dust. The aura region analysis may include calculating 
effects of differences in values of the one or more extracted parameters in different 
images of dust artifact illumination, shape, position, reflection or transmission 
properties, distance of dust to the sensor, aperture, exit pupil, or focal length, or 
combinations thereof. The different images may be acquired with different values of 

15 the one or more extracted parameters, and the different images may be acquired of 
different objects. 

The method may further include validating whether a further digitally-acquired 
image has non-contradicting data that the probability that certain pixels correspond 
to dust artifact regions within the image prior to correcting pixels corresponding to 

20 correlated dust artifact regions within the images based on the associated statistical 
dust map. The user may be instructed to create a new dust map if the validating 
determines a noticeable change in the dust map. The shadow region analysis may 
include calculating effects of differences in values of the one or more extracted 
parameters in different images on dust artifact illumination, shape, position, reflection 

25 or transmission properties, distance of dust to the sensor, aperture, exit pupil, or 
focal length, or combinations thereof. 

Suspected dust artifact regions of the further digitally images may include 
inner or shadow regions and aura regions. Focal length extracted parameters may 
be calculated empirically from comparison of the transposition of inner or shadow 

30 regions of dust artifact regions within multiple original digital images acquired with 
the digital acquisition device. Aperture extracted parameters may be calculated 
empirically from comparison of fall off of aura regions of the dust artifact regions 
within the multiple original digital images acquired with the digital acquisition device. 
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The correcting may include a first correcting of the aura regions and a second 
correcting of the inner regions, in either order. 

The method may be performed on raw image data as captured by a camera 
sensor. The method may be performed on a processed image after being converted 
5 from raw format to a known red, green, blue representation. The correcting 

operation may include replacing pixels within one or more digitally-acquired images 
with new pixels. The correcting may include enhancing values of pixels within one or 
more digitally-acquired images. Correcting instructions may be kept in an external 
location, such as an image header, to the image data. 

10 The dust artifact probability determining operation may include loading the 

statistical dust map, loading extracted parameter information of a present image, 
performing calculations within the statistical dust map having extracted parameter 
variable-dependencies, and comparing dust artifact detection data with the extracted 
parameter dependent statistical dust map data. The extracted parameter information 

15 may include values of aperture size, focal length and/or lens type information. The 
dust artifact probability determining operation may also include loading the statistical 
dust map, loading extracted parameter information of a present image, performing a 
calculation for relating the statistical dust map with the present image according to a 
selected value of an extracted parameter which is otherwise uncorrelated between 

20 the present image and the dust map, and comparing dust artifact detection data with 
the now correlated statistical dust map data. 

Suspected dust artifact regions of at least two images may include inner or 
shadow regions and aura regions. The comparison may include a comparison of the 
inner regions and a different comparison of the aura regions, in either order. The 

25 dust artifact regions may include an aura region partially obscured by dust and a 
shadow region substantially obscured by dust inside the aura region. 

The determining with respect to a shadow region may be based on an 
extracted parameter dependent shadow region analysis, wherein the shadow region 
analysis presumes that certain regions on a sensor of the digital image acquisition 

30 device are substantially obscured by said dust. The shadow region analysis may 
include calculating effects of differences in values of the one or more extracted 
parameters in different images on dust artifact illumination, shape, position, reflection 
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or transmission properties, distance of dust to the sensor, aperture, exit pupil, or 
focal length, or combinations thereof. 

The determining with respect to an aura region may be based on an extracted 
parameter dependent aura region analysis, wherein the aura region analysis 
5 presumes that certain regions on a of the digital image acquisition device are 
partially obscured by said dust. The aura region analysis may include calculating 
effects of differences in values of the one or more extracted parameters in different 
images on dust artifact illumination, shape, position, reflection or transmission 
properties, distance of dust to the sensor, aperture, exit pupil, or focal length, or 

10 combinations thereof. 

The correcting operation may include in-painting or restoration, or both. The 
in-painting correcting may be applied to the shadow region. This in-painting may 
include determining and applying shadow region correction spectral information 
based on spectral information obtained from pixels outside the shadow region. The 

15 restoration correcting may be applied to the aura region. The restoration may 

include determining and applying aura region correction spectral information based 
on spectral information obtained from pixels within the aura region. 

In addition, in methods that may be performed according to preferred 
embodiments herein, the operations have been described in selected typographical 

20 sequences. However, the sequences have been selected and so ordered for 
typographical convenience and are not intended to imply any particular order for 
performing the operations, unless a particular ordering is expressly provided or 
understood by those skilled in the art as being necessary. 

Many references have been cited above herein, and in addition to that which 

25 is described as background, the invention summary, brief description of the 

drawings, the drawings and the abstract, these references are hereby incorporated 
by reference into the detailed description of the preferred embodiments, as 
disclosing alternative embodiments of elements or features of the preferred 
embodiments not otherwise set forth in detail above. A single one or a combination 

30 of two or more of these references may be consulted to obtain a variation of the 
preferred embodiments described in the detailed description above. 
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What is claimed is: 

Claim 1: A digital image acquisition device, comprising 

(a) an image acquisition sensor, 

(b) a shutter, 

5 (c) optical elements for focusing an image on the sensor, 

(d) an aperture disposed between said optical elements and said shutter, 

(e) a translucent lens cap which defocuses images when in place, and 

(f) digitally-embedded programming such that the device is arranged to 
automatically capture a calibration image of a de-focused scene when said shutter 

10 and said aperture are open, and to derive a map of defects from the calibration 
image acquired by the sensor when illuminated by ambient light. 

Claim 2: The device of claim 1 , further comprising a trigger for automatically 
acquiring a de-focused calibration image which is arranged to automatically detect 
15 whether the translucent lens cap is in place as a pre-requisite for triggering capture 
of the defocused calibration image. 

Claim 3: The device of claim 2, wherein the trigger is arranged to trigger said 
capture based at least in part on time interval, change of lens, or amount of images 
20 taken, or a combination thereof. 

Claim 4: A digital image acquisition device having no photographic film, comprising 
an optical system, a processor and one or more storage devices for programming 
the processor to control the camera to perform a method of automatically correcting 
25 dust artifact regions within acquired images, the method comprising: 

(a) acquiring one or more original digital images through a translucent lens 
cap disposed along an optical path between an object and one or more components 
of the optical system including a digital sensor; 

(b) determining probabilities that certain pixels correspond to dust or other 
30 blemish artifact regions within said one or more original digital images; 
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(c) associating the dust or other blemish artifact regions with one or more 
extracted parameters relating to the optical system when the one or more original 
digital images were acquired; 

(d) forming a statistical dust map including mapped dust or other blemish 
5 regions based on the dust or other blemish artifact probability determining and 

associating; and 

(e) correcting pixels corresponding to dust artifact regions within each of 
said one or more original images based on the associated statistical dust map. 

10 Claim 5: The device of claim 4, wherein said one or more original images comprise 
original images of de-focused scenes. 

Claim 6: The device of claim 4, wherein said determining probabilities further 
comprises statistically combining a plurality of individual probabilities based on each 
15 said region within two or more said images. 

Claim 7: The device of claim 4, wherein the method further comprises determining 
probabilities that certain pixels correspond to regions free of dust or other blemishes 
within said images based at least in part on a comparison of suspected dust artifact 
20 regions within one or more of said images. 

Claim 8: The device of claim 4, wherein the method further comprises eliminating 
certain suspected dust or other blemish artifact regions as having a probability below 
a first threshold value. 

25 

Claim 9: The device of claim 4, wherein the method further comprises judging 
certain probable dust or other blemish artifact regions as having a probability above 
a second threshold value such as to be subject to said correcting. 

30 Claim 10: The device of claim 4, wherein said one or more extracted parameters 
comprise aperture size or focal length or both. 
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Claim 1 1 : The device of claim 4, wherein the method further comprises digitally- 
acquiring additional images with said digital acquisition device, repeating said 
determining and associating, and updating said statistical dust map including 
updating said mapped dust or other blemish regions based on the additional dust or 
5 other blemish artifact determining and associating. 

Claim 1 2: The device of claim 1 1 , wherein said determining includes dynamically 
updating said probabilities based on comparisons with suspected equivalent dust 
artifact regions within the additional digitally-acquired images. 

10 

Claim 1 3: The device of claim 4, the dust or other blemish artifact determining 
including: 

(I) loading the statistical dust map; 

(II) loading extracted parameter information of a present image; 

15 (III) performing calculations within the statistical dust map having extracted 

parameter variable-dependencies; and 

(IV) comparing dust artifact detection data with the extracted parameter 
dependent statistical dust map data. 

20 Claim 14: The device of claim 4, the dust or other blemish artifact determining 
including: 

(I) loading the statistical dust map; 

(II) loading extracted parameter information of a present image; 

(III) performing a calculation for relating the statistical dust map with the 
25 present image according to a selected value of an extracted parameter which is 

otherwise uncorrelated between the present image and the dust map; and 

(IV) comparing dust artifact detection data with the now correlated statistical 
dust map data. 

30 Claim 15: The device of claim 4, wherein the method further comprises sorting dust 
or other blemish artifact distribution data within said dust map according to meta- 
data. 
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Claim 16: The device of claim 4, wherein said digital acquisition device captures 
said images from film images. 

Claim 17: A digital image acquisition device, comprising 
5 (a) an image acquisition sensor, 

(b) a shutter, 

(c) optical elements for focusing an image on the sensor, 

(d) an aperture disposed between said optical elements and said shutter, 

(e) a lens which defocuses images when in a dust-detection mode, and 
10 (f) digitally-embedded programming such that the device is arranged to 

automatically capture a calibration image of a de-focused scene when said shutter 
and said aperture are open, and to derive a map of defects from the calibration 
image acquired by the sensor when illuminated by ambient light. 

15 Claim 18: The device of claim 17, wherein the trigger includes programming for 

analyzing whether the image is sufficiently de-focused as a pre-requisite for capture 
of the de-focused calibration image. 

Claim 19: The device of claim 17, wherein the trigger is arranged to initiate capture 
20 of a plurality of de-focused calibration images of the scene within a certain time 
frame. 

Claim 20: The device of claim 17, further comprising validating that said additional 
digitally-acquired images have non-contradicting probability data that certain pixels 
25 correspond to dust or other blemish artifact regions within said additional digitally- 
acquired images prior to correcting pixels corresponding to correlated dust artifact 
regions within additional digitally-acquired images based on the associated statistical 
dust map. 
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